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Introduction 

“American education will never realize its potential as an engine of opportunity and 
economic growth until a writing revolution puts language and communication in their proper 
place in the classroom.” Thus begins the call for action articulated by the National Commission 
on Writing in America’s Schools and Colleges (2003, p. 3). Created by the College Board (a 
nonprofit membership organization composed of more than 4,300 schools and colleges), the 
commission is composed of academic experts on writing, teachers, superintendents, and college 
and university presidents and chancellors. “The Neglected ‘R’ - The Need for a Writing 
Revolution” advocates for sweeping changes in educational policy including a comprehensive 
writing policy; preservice and inservice training for all teachers; federal and state funding; a 
doubling of time spent on writing in schools; writing across the curriculum; authentic assessment 
of writing competence; alignment of standards, curriculum, and assessment; technology; and 
university-school partnerships. 

“What most students cannot do is write well. At least, they cannot write well enough to 
meet the demands they face in higher education and the emerging work environment” (National 
Commission on Writing in America’s Schools and Colleges, 2003, p. 16). Citing statistics from 
the 1998 National Assessment of Educational Progress, it is clear that achievement of writing 
competencies should be of concern. Twenty-three percent of fourth graders achieved a proficient 
level, while 27% and 22% of eighth graders and twelfth graders, respectively met this criterion. 
Across all grade levels, 1% of students achieved an “advanced” level of proficiency in writing 
(National Center for Education Statistics, 1999). As a point of comparison, in reading 
achievement, 39% of the nation’s fourth graders scored at the proficient or advanced level (8% 
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advanced). In mathematics, 36% of fourth graders scored at the proficient or advanced level (4% 
advanced) - an increase of 22% since 1990. 

Some cause for optimism in writing achievement was noted from the 2002 NAEP 
Writing Assessment. Mean scale scores for fourth and eighth grade students showed significant 
increases. Fourth grade mean scale scores rose from 150 in 1998 to 154 in 2002, while eighth 
grade scores improved from 150 to 153 (twelfth graders declined from 150 to 148). These scale 
scores can be translated into percentages of students scoring at proficient or advanced levels of 
28% in fourth grade, 31% in eighth grade, and 24% in twelfth grade (National Center for 
Education Statistics, 2003). 

However, the long-term patterns of writing achievement have indicated similar declines 
and improvements since 1984, as indicated in Figure 1. 
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Figure 1. National Assessment of Educational Progress Writing Assessment Results 
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From this data, three major conclusions may be drawn. Student writing achievement is 
the most depressed major skill area for students in the United States, and has been so since 1984. 
Two-thirds of American public school students fail to achieve proficiency in writing during their 
K-12 school years. Finally, there are no clear patterns of improvement in the last 20 years. If, as 
the National Commission on Writing in America’s Schools and Colleges asserts, writing is a 
fundamental skill necessary for full participation in our economic, social, and political systems, 
then an educational discourse nearing the magnitude of the “reading wars” (Reyhner, 2008) 
needs to begin today. 

This country’s stagnant writing achievement profile might lead one to believe that writing 
instruction in the United States has been stagnant as well. However, we have already 
experienced our first “writing revolution” - a progressivist movement begun during the Vietnam 
War era. It was sparked by concerns that most students would never learn to write in 
“traditional” school environments characterized by authoritarian classroom structures and 
meaningless grammar drill and practice (Murray, 1969). The movement soon became known as 
“process writing.” 

In the United States Department of Education’s report, What Happens in Classrooms? 
Instructional Practices in Elementary and Secondary Schools, 1994-95, more than two-thirds of 
elementary school teachers reported that “the writing process” (a.k.a., writer’s workshop, process 
writing) was the primary instructional approach used for writing instruction. 

Fumeaux (1998) provides an important conceptualization of writing theory. The process 
writing movement is based in a cognitive process theory of writing (Flower and Hayes, 1981). 

In this theory, writing is complex, goal-directed, recursive activity in which students progress 
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from “production of egocentric, writer-based texts (typically, writing everything they know on a 
topic without thinking of what the reader wants or needs to know) to reader-based texts, which 
are written with the reader in mind” (Furneaux, p. 258). “This model has been criticized for 
being too vague (with no reference to how text is actually produced) or too generalized (the 
model suggests a uniform process for all writers)” (p. 259). 

In response to these criticisms, social stage theory appeared in the form of Michael 
Halliday’s “systemic functional linguistics” work at the University of Sydney, Australia 
(Halliday, 1973, 1974, 1975, 1976, 1977, 1985). Halliday and his contemporaries (Christie and 
Martin, 1997; Cope & Kalantzis, 1993; Painter & Martin, 1986; Benson, Cummings & Greaves, 
1984) postulated that all communication, written or oral, occurs in the dimension of a social 
context. Therefore, the elements of writer’s purpose and audience needs/expectations are central 
to the process. These ideas became collectively known as the genre-based approach (Cope & 
Kalantzis, 1993) - for twenty years the dominant model of writing instruction in Australian 
public schools. 

Review of the Literature 

The Rise of Process Writing 

In 1968, Peter Elbow proposed that self-expression through “freewriting” (non-directed 
writing) is a “prior achievement in the process of learning to write well” (p. 115). Elbow noted 
that, “True writing and good prose are only end products rather than the primary objectives” (p. 

1 15) in his proposed method for the writing instruction of college freshmen. Freewriting is the 
vehicle by which writer’s block can be avoided; it develops the author’s voice and allows a 
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process for “cooking” ideas. In short, the process was as or more important than the written 
product itself. 

In 1969, Pulitzer Prize winning journalist Donald Morrison Murray (then a college 
professor of composition) extended the concept of the “writing process” to elementary education. 
In general he advocated for the teaching of a cyclical process of prewriting, writing, and 
rewriting, which, as he noted, is practiced by the publishing author” (p. 4). “By becoming aware 
of his world and gathering the raw specifics that will be paragraphs, the student can begin to find 
his subject, select his information, choose his vehicle of expression, identify the audience, and 
develop a feeling for the appropriate structure, tone, and point of view” (p. 3). Through 
application of Elbow’s “freewriting” procedures and regular student-teacher conferencing, he 
proposed that student writing would improve. 

Another founding voice in this movement was Ken Macrorie, who had studied alongside 
Peter Elbow. He similarly advocated for classrooms wherein students were free to explore their 
own ideas. Freewriting could allow writers to develop the own voices; therefore, grammar 
should not be a primary consideration in the writing process. The theories, scholarly writing, and 
research of Elbow, Murray, and Macrorie formed the foundations for a new movement in writing 
instruction that has become known as “process writing” - an approach that has sustained as the 
dominant force in writing practices since the early 1970s (Reppen, 1994). 

Considered the “father of process writing” in elementary education, Donald H. Graves 
became the preeminent authority in elementary writing instruction 1983 with the publication of 
Writing: Teachers & Children at Work. Applying the processes of prewriting, writing, and 
revision, Graves began the work of making this philosophy accessible to classroom teachers. 



Every Child a Writer 2014 


7 

Since that time, he has published more than a dozen additional texts discussing various forms of 
writing, classroom management, conferencing, handwriting, spelling, and language development. 

The prevalence of the writing process as a primary instructional approach in elementary 
schools is documented in the United States Department of Education ’s report, What Happens in 
Classrooms? Instructional Practices in Elementary and Secondary Schools, 1994-95. Fifty- 
seven percent of primary grades teachers (K-3) reported engaging students in “self-reflective” in 
the semester immediately preceding the teacher survey. Further, 74% of all primary teachers 
used portfolio-type assessments for language arts instruction. 

Goldstein and Carr (1996) analyzed teacher and student survey responses from the 1992 
National Assessment of Educational Progress Writing Assessment to determine the frequency 
with which teachers were implementing writing process activities in the classroom. The authors 
found that students “who reported being always asked to do certain elements of a structured 
approach to writing had higher average NAEP scores than those who reported never being asked 
to do them” (p. 2). Specifically, students who were consistently asked to plan their writing, 
define their purpose and audience, to use information sources beyond the textbook, or to write 
more than one draft of a paper scored higher than students never asked to do these activities. 
Students who engaged in a greater number of these activities on a regular basis scored higher 
than those who engaged in fewer activities. Further, students who used lists, diagrams, or 
outlines as planning tools scored higher than students who used other techniques. While these 
factors influenced students’ achievement on writing assessments, the degree of difference in 
percentage of proficiency in the population was less than 3%. Clearly, the writing process, 
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consistently applied can affect student achievement, although the gains over “traditional” 
methods were modest, at best. 

Genre-Based Writing 

The theory of genre-based writing grew from an educational experiment that began in 
Sydney, Australia in the early 1980s. “By the 1980s it was clear that the progressivist 
curriculum [process writing instruction] was not producing the goods. It was not producing any 
noticeable improvement in patterns of educational attainment. In fact, all it seemed to do was 
make teachers’ jobs harder” (Cope & Kalantzis, 1993, p. 1). 

The genre theorists asserted that “genre” is based in the social, communicative purposes 
of texts, and that these purposes had grown significantly different and more complex in the latter 
quarter of the twentieth century due to the rise of technology and communication of information 
(Kalantzis and Wignell, 1988). Further, the ability to communicate for multiple social purposes 
and audiences gives their creators access to realms of social influence and power (Kress, 1989). 
Typically, marginalized populations (e.g., ethnic minorities, students of diverse language 
backgrounds, children of poverty) have been denied access to power and influence due to less 
developed communication skills. 

Every Child a Writer (ECAW) is a school-wide model of comprehensive writing 
instruction and assessment (Johnson & Rochester, 2000). The model is based on the 
foundational structures and principles of Australian genre-based approach, which was imported 
and adapted by the curriculum and program development at the National Literacy Coalition 


(NLC). 
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Periodically, the NLC requests a review of ECAW Member Schools’ performance on 
state writing assessments. As one element of this analysis, these schools’ assessment results are 
compared to the state average for the same group of students. This paper documents these 
results from ECAW-implementing elementary schools in Colorado, where the model has been 
used since 2000. 

The schools selected for the study include all ECAW “Member Schools” in Colorado. 
NLC defines a “member school” as a school that has provided all instructional staff complete 
with the required ECAW professional development seminar, a one credit-hour course. Each 
school also assures that 100% of instructional staff have access to the ECAW instructional 
resources (either in paper or digital editions). 

Research Design 

The four research questions guiding the design of the study were as follows: 

• Research Question 1 . How did ECAW schools perform relative to the state average? 

• Research Question 2. Did ECAW schools demonstrate gains, remain flat, or decline 
in performance over time? 

• Research Question 3. Were the ECAW schools’ gain rates (if applicable) at a level 
equal to, below, or above the state average? 

• Research Question 4. Were the ECAW schools’ gain scores (if applicable) 
statistically significant when compared to the state average? 

The source of the data used in this analysis is the Colorado Department of Education. 
Each year, the department releases school and district data spreadsheets documenting student 
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performance on the CSAP/TCAP writing assessments. These documents are available to the 
general public on the department’s website. 

A cohort group design was selected as the method of data analysis for this study for 
several reasons. First, the State of Colorado assesses all students in grades 3, 4, 5, and 6 
annually; this allows for comparisons of students on the same scale over a four-year period, thus 
allowing reasoned inferences regarding school growth over time. Secondly, in 2010, Colorado 
adopted a growth model for measuring school progress. Finally, the number of Every Child a 
Writer Member Schools increases annually. This allows for data analysis among “cohort school 
groups” (schools that became members in the same year), and, within that data set, multiple 
student cohort groups. Specifically, the following Cohort School Groups and student cohort 
groups were analyzed. 

Cohort School Group 1 

• Nineteen schools 

• Began implementation in 2007 or before 

• Four student cohort groups (2007-2010, 2008-2011, 2009-2012, and 2010-2013) 
Cohort School Group 2 

• Nine schools (three schools were not included due to lack of sufficient data) 

• Began implementation in 2009 (baseline data from 2008 included) 

• Four student cohort groups (2008-20 1 0, 2009-20 11,201 0-20 1 2, and 2011 -20 13) 
Cohort School Group 3 


• Seventeen schools 
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• Began implementation in 2010 (baseline data from 2009 included) 

• Three student cohort groups (2009-2011, 2010-2012, and 2011-2013) 

Data Analysis Procedure 

The State of Colorado uses a five-level achievement reporting system. These levels 
include “unsatisfactory,” “partially proficient,” “proficient,” “advanced,” and “proficient and 
advanced.” While the data was analyzed for each of these levels, for the purposes of this 
analysis, the three categories of “unsatisfactory,” “proficient and advanced,” and “advanced” are 
reported. Figure 1 shows a sample data collection sheet. “State totals” was entered from 
Colorado Department of Education spreadsheets (row 1). EC AW school data was extracted from 
the same spreadsheets and included in row 2. ECAW data was subtracted from the state totals 
data. Percentages in each performance category were generated by dividing the numbers in the ( 
# ) columns and dividing by the total students. 


School 

Total 

Students 

Unsatisfactory 

Partially 

Proficient 

Proficient 

Advanced 

Pro & 
Adv 
% 

# 

% 

# 

% 

# 

% 

# 

% 

STATE TOTALS 

61,172 

4074 

7 

25880 

42 

25614 

42 

5202 

9 

50 

ECAW CH2 

254 

8 

3.15 

91 

35.83 

124 

48.82 

31 

12.20 

61.02 

STATE - ECAW 

60,918 

4,066 

6.67 

25,789 

42.33 

25,490 

41.84 

5,171 

8.49 

50.33 


Figure 2. Sample data analysis sheet 


To test for the statistical significance of the results, the t-test procedure was used to 
analyze each cohort’s data set. To determine the mean for the group, point values were assigned 
based on each student’s performance level on the CSAP/TCAP. An “unsatisfactory” score 
received a point value of 1 . A “partially proficient” score received 2 points, a “proficient” score 
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received “3,” and “advanced” was assigned a point value of “4.” These results were added, and 
the mean and standard deviations were calculated. For the t-test analysis, a 95% confidence 
level was set. The p values were then calculated for each cohort data set. 

Results 

The data for three Cohort Schools Groups, including multiple student cohort groups is 
presented in the next several pages. Cohort Schools Group 1 includes four student cohorts (the 
third through sixth grade students in each of four successive years beginning in 2007). Cohort 
School Group 1 is made up of Bennett ES, Burlington ES (RE-6J); Cherokee Trail ES, Eldorado 
ES, Heritage ES, Iron Horse ES, Larkspur ES, Legacy Point ES, Mountain View ES, Northridge 
ES, Pine Grove ES, Saddle Ranch ES, Sand Creek ES, Timber Trail ES, Trailblazer ES 
(Douglas); Prairie Winds ES (Lewis-Palmer 38); Prairie Winds ES (Pueblo 70), Wiggins ES, and 
William E Bishop ES (Englewood). 

Three graphs are presented for each student cohort group: percentage of students 
achieving proficient and advanced scores, percentage achieving advanced scores, and percentage 
in the unsatisfactory performance level. Additionally, the gain score for each group is identified. 
The gain score was calculated by subtracting the group’s mean baseline percentage from the final 
grade level’s mean percentage. As a measure of the goal of reducing the percentage of students 
in the unsatisfactory performance level, these changes are reported as the mean percent of the 
decline in this population. The formula for this statistic is the mean of the baseline grade level 
unsatisfactory percentage (b) subtracted from the mean of the group’s final grade level 
unsatisfactory percentage (f), the result then divided by the group’s final grade level 
unsatisfactory percentage (f), or (f - b) / f. 
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School Cohort Group 1: Student Cohort Group 1 (2007-2010) 



Figure 3. School Cohort Group 1, Student Cohort 1: Percent Proficient and Advanced 
Pre-Post Gains: ECAW8.41. Colorado 2.82. 



Figure 4. School Cohort Group 1 , Student Cohort 1 : Percent Advanced 
Pre-Post Gains: ECAW4.78. Colorado 1.10. 



Figure 5. School Cohort Group 1, Student Cohort 1: Percent Unsatisfactory 
Pre-Post Change: ECAW -77.21%. Colorado -15.17% 
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School Cohort Group 1 : Student Cohort Group 2 (2008-2011) 



Figure 6. School Cohort Group 1, Student Cohort 2: Percent Proficient and Advanced 
Pre-Post Gains: ECAW 13.95. Colorado 11.89. 



Figure 7. School Cohort Group 1, Student Cohort 2: Percent Advanced 
Pre-Post Gains: ECAW 5.89. Colorado 0.55. 



Figure 8. School Cohort Group 1, Student Cohort 2: Percent Unsatisfactory 
Pre-Post Change: ECAW -64.57%. Colorado -38.94% 
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School Cohort Group 1: Student Cohort Group 3 (2009-2012) 



Figure 9. School Cohort Group 1, Student Cohort 3: Percent Proficient and Advanced 
Pre-Post Gains: ECAW 9.09. Colorado 4.07. 



Figure 10. School Cohort Group 1, Student Cohort 3: Percent Advanced 
Pre-Post Gains: ECAW 6.25. Colorado -0.38. 



Figure 11. School Cohort Group 1, Student Cohort 3: Percent Unsatisfactory 
Pre-Post Change: ECAW -57.89%. Colorado -27.94% 
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School Cohort Group 1: Student Cohort Group 4 (2010-2013) 



Figure 12. School Cohort Group 1, Student Cohort 4: Percent Proficient and Advanced 
Pre-Post Gains: ECAW9.74. Colorado 7.32. 



Figure 13. School Cohort Group 1, Student Cohort 4: Percent Advanced 
Pre-Post Gains: ECAW8.61. Colorado 0.22. 



Figure 14. School Cohort Group 1, Student Cohort 4: Percent Unsatisfactory 
Pre-Post Change: ECAW -59.98%. Colorado -37.62%. 
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In 2009, Cohort Schools Group 2 became ECAW Member Schools. The baseline 
performance year for each of these schools was 2008. These schools included Bethune ES 1 (Kit 
Carson County in eastern Colorado); Cotopaxi ES 1 (Fremont County in south central Colorado); 
Fraser Valley ES, Granby ES, Grand Lake ES 1 (East Grand School District, Grand County, in 
north central Colorado); Leawood ES (Jefferson County, Littleton, in the southwestern Denver 
metropolitan area) Parmalee ES (Jefferson County, Indian Hills, in the foothills west of Denver) 
and Ute Meadows ES (Jefferson County, Littleton). The Colorado Department of Education, in 
its publicly released data, does not include school grade levels in which the enrollment is fewer 
than 16 students. Schools denoted with the superscript 1 were not included in the analysis due to 
this policy. 

The majority of the schools in Cohort Schools Group 2 feature a K-5 grade level 
configuration. In fact, within this group of schools, only two, Leawood ES and Lite Meadows ES 
(Jefferson County), counted sixth grade students in their enrollment figures. Including the sixth 
grade data from these schools falsely skews the data trends upward; therefore, these sixth grade 
classes were excluded from this analysis. 

Four grades 3-5 student cohort groups are reported in the next several pages. These 
include the third through fifth grade classes of 2008-2010, 2009-2011, 2010-2012, and 2011- 


2013. 
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School Cohort Group 2: Student Cohort Group 1 (2008-2010) 



Figure 15. School Cohort Group 2, Student Cohort 1: Percent Proficient and Advanced 
Pre-Post Gains: ECAW 13.80. Colorado 7.24. 



Figure 16. School Cohort Group 2, Student Cohort 1: Percent Advanced 
Pre-Post Gains: ECAW 2.79. Colorado -0.87. 



Figure 17. School Cohort Group 2, Student Cohort 2: Percent Unsatisfactory 
Pre-Post Change: ECAW -32.02%. Colorado -21.74%. 
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School Cohort Group 2: Student Cohort Group 2 (2009-2011) 



Figure 18. School Cohort Group 2, Student Cohort 2: Percent Proficient and Advanced 
Pre-Post Gains: ECAW 12.68. Colorado 6.62. 



Figure 19. School Cohort Group 2, Student Cohort 2: Percent Advanced 
Pre-Post Gains: ECAW 7.97. Colorado 1.36. 



Figure 20. School Cohort Group 2, Student Cohort 2: Percent Unsatisfactory 
Pre-Post Change: ECAW -66.05%. Colorado -27.35. 
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School Cohort Group 2: Student Cohort Group 3 (2010-2012) 



Figure 21. School Cohort Group 2, Student Cohort 3: Percent Proficient and Advanced 
Pre-Post Gains: ECAW 10.95. Colorado 8.05. 



Figure 22. School Cohort Group 2, Student Cohort 3: Percent Advanced 
Pre-Post Gains: ECAW 10.39. Colorado 2.31. 



Figure 23. School Cohort Group 2, Student Cohort 3: Percent Unsatisfactory 
Pre-Post Change: ECAW -33.58%. Colorado -22.1 1%. 
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School Cohort Group 2: Student Cohort Group 4 (2011-2013) 



Figure 24. School Cohort Group 2, Student Cohort 4\ Percent Proficient and Advanced 
Pre-Post Gains: ECAW 6.54. Colorado 5.88. 



Figure 25. School Cohort Group 2, Student Cohort 4\ Percent Advanced 
Pre-Post Gains: ECAW 5.85. Colorado 0.075. 



Figure 26. School Cohort Group 2, Student Cohort 4: Percent Unsatisfactory 
Pre-Post Change: ECAW +2.86%. Colorado -17.23%. 
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In 2010, Cohort Schools Group 3 began implementing ECAW. The following schools 
became ECAW Member Schools: East ES, Maybell ES 1 , Ridgeview ES, Sandrock ES, Sunset 
ES (Moffat County RE-1 in the Craig area of Northwestern Colorado); Bergen Valley IS, Elk 
Creek ES, Marshdale ES, West Jefferson ES, Wilmot ES (Jefferson County schools in the 
foothills west of Denver); Fort Logan ES (Sheridan School District in the southwestern Denver 
metropolitan area); Platte Valley ES 1 (Platte Valley RE-3 in Sedgwick County in extreme 
Northeastern Colorado); Buffalo Ridge ES, Prairie Crossing ES, Rock Ridge ES (Douglas 
County schools, south Denver metropolitan region); Prairie ES 1 (Prairie Re-1 1 J in Weld County 
in Northeastern Colorado); Sierra Grande ES 1 (Sierra Grande R-30, Costilla County in south 
central Colorado); Julesburg ES 1 (Julesburg RE-1, Sedgwick County in extreme northeastern 
Colorado); Meeker ES (Meeker RE-1, Rio Blanco County in west central Colorado); and Wray 
ES (Wray RD-2, Yuma County in northwestern Colorado). 

Within this Cohort Schools Group, only two, Prairie Crossing ES and Rock Ridge ES, 
maintained sixth grade students in their enrollment figures. This small number of students 
falsely skewed the sixth grade upward; therefore, the sixth grade data is not included in this 
analysis. Three student cohort groups are documented below. These include the third through 
fifth grade classes of 2009-2011, 2010-2012, and 201 1-2013. 
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School Cohort Group 3: Student Cohort Group 1 (2009-2011) 



Figure 27. School Cohort Group 3, Student Cohort 1 : Percent Proficient and Advanced 
Pre-Post Gains: ECAW4.52. Colorado 6.69. 



Figure 28. School Cohort Group 3, Student Cohort 1: Percent Advanced 
Pre-Post Gains: ECAW0.98. Colorado 1.40 



Figure 29. School Cohort Group 3, Student Cohort 1: Percent Unsatisfactory 
Pre-Post Change: ECAW -36.29%. Colorado -27.41. 
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School Cohort Group 3: Student Cohort Group 2 (2010-2012) 



Figure 30. School Cohort Group 3, Student Cohort 2: Percent Proficient and Advanced 
Pre-Post Gains: ECAW9.71. Colorado 8.04. 



Figure 31. School Cohort Group 3, Student Cohort 2: Percent Advanced 
Pre-Post Gains: ECAW 3.3. Colorado 2.33. 



Figure 32. School Cohort Group 3, Student Cohort 2: Percent Unsatisfactory 
Pre-Post Change: ECAW -16.19%. Colorado -22.24%. 
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School Cohort Group 3: Student Cohort Group 3 (2011-2013) 



Figure 33. School Cohort Group 3, Student Cohort 3: Percent Proficient and Advanced 
Pre-Post Gains: ECAW 9.32. Colorado 5.84. 



Figure 34. School Cohort Group 3, Student Cohort 3: Percent Advanced 
Pre-Post Gains: ECAW 2.67. Colorado 0.75 



Figure 35. School Cohort Group 3, Student Cohort 3: Percent Unsatisfactory 
Pre-Post Change: ECAW -20.38%. Colorado -17.19. 
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The mean gain scores (in percentage of students achieving proficient and advanced 
performance levels) are reported in Figure 36. Across the 11 student cohorts, ECAW schools 
achieved a mean gain over baseline of 9.883, while the mean gain for Colorado was 6.777. The 
difference of 3.106 was found to be statistically significant (p < .0122). 


15 



■ ECAW ■ Colorado 


Figure 36. All Cohort Groups Gains over Baseline - Proficient and Advanced 

The mean gain scores (in percentage of students achieving the advanced performance level) are 
reported in Figure 37. Across the 1 1 student cohorts, ECAW schools achieved a mean gain over 
baseline of 5.407, while the mean gain for Colorado was 0.866. The difference of 4.541 was 
found to be statistically significant (p < .0004). 
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■ ECAW ■ Colorado 


Figure 37. All Cohort Groups Gains over Baseline - Advanced 
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The mean reduction scores (percentage of students achieving the unsatisfactory performance 
level) are reported in Figure 38. Across the 1 1 student cohorts, ECAW schools achieved a mean 
reduction from baseline of 1.884, while the mean reduction for Colorado was 1.201. The 
difference of 0.683 was not statistically significant {p < .3194). 
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Figure 38. All Cohort Groups Percent Reductions from Baseline - Unsatisfactory 
Mean Reduction from Baseline: ECAW 49.22%. Colorado 24.84% 


The data table in Figure 39 presents the mean baseline and mean post (final grade level of 
implementation) data for each of the 1 1 student cohorts as well as the gain score differences for 
each cohort group. The mean baseline assessment levels for the 1 1 cohorts was 0.1792 for the 
ECAW schools. Baseline assessment levels for the Colorado cohorts was 0.0947. The 
difference between these means was 0.0845. Using a t-test analysis, the mean difference 
between groups was found to be statistically significant (p < .0014). 
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ECAW 

Baseline 

m 

ECAW 

Baseline 

(SD) 

ECAW Post 
(M) 

ECAW Post 
(SD) 

ECAW Gain 

over 

Baseline 

Colorado 

Baseline 

(M) 

Colorado 

Baseline 

(SD) 

Colorado 

Post 

(M) 

Colorado 

Post 

(SD) 

Colorado 
Gain over 
Baseline 

Gain 

(difference) 

Cohort 1 

2.819 

0.679 

2.974 

0.6340 

0.1550 

2.569 

0.7294 

2.627 

0.7623 

0.0580 

0.0970 

Cohort 2 

2.821 

0.7338 

3.106 

0.6434 

0.2850 

2.532 

0.768 

2.685 

0.7179 

0.1530 

0.1320 

Cohort 3 

2.829 

0.6778 

3.031 

0.6413 

0.2020 

2.565 

0.7379 

2.621 

0.7034 

0.0560 

0.1460 

Cohort 4 

2.806 

0.6765 

3.025 

0.6749 

0.2190 

2.519 

0.745 

2.621 

0.7030 

0.1020 

0.1170 

Cohort 5 

2.71 

0.7537 

2.89 

0.7047 

0.1800 

2.539 

0.7685 

2.621 

0.7298 

0.0820 

0.0980 

Cohort 6 

2.693 

0.7276 

2.93 

0.6986 

0.2370 

2.571 

0.7376 

2.668 

0.7191 

0.0970 

0.1400 

Cohort 7 
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Cohort 8 
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Cohort 1 1 

2.647 
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0.7197 

0.0800 

0.0520 

Means 

2.739 


2.919 


0.1792 

2.544 
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0.0947 
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Figure 39. ECAW Member School and Colorado results: pre- and post-means (M) and gains over 
baseline. 


Effect size (r) was calculated based on these results. A positive effect size of 0.6532 was 

found. 



All cohorts gain 
scores (M) 

All cohorts (SD) 

Significance (p) 

Effect-size (r) 

ECAW 

0.1792 

0.0624 

< .0014 

0.6532 

Colorado 

0.0947 

0.0282 


Figure 40. Figure 41. Comparing ECAW Member Schools and Colorado gain scores ( M) and measure of 
statistical significance (p-value) 
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Discussion 

The goal of the current study was to answer, in detail, the following research questions: 

• Research Question 1 . How did ECAW schools perform relative to the state average? 

• Research Question 2. Did ECAW schools demonstrate gains, remain flat, or decline 
in performance over time? 

• Research Question 3. Were the ECAW schools’ gain rates (if applicable) at a level 
equal to, below, or above the state average? 

• Research Question 4. Were the ECAW schools’ gain scores (if applicable) 
statistically significant when compared to the state average? 

As a group, the eleven student cohorts in this study performed significantly above the 
state average, both in mean scores as well as in the specific performance categories of proficient, 
advanced, and unsatisfactory. The mean achievement of the ECAW schools was 2.919 as 
compared to the state mean of 2.639. The difference in means (0.28) was statistically significant 
at the p < .0001 level. 

The ECAW student cohorts demonstrated gains (growth) at a level above the state gain 
rate. Colorado’s mean pre- versus post-test gain score was 0.0947. For the same group and 
time, ECAW schools gain score was 0.1792. The difference in gain scores between groups was 
significant at the p < .0014 level. 

Effect size (r) was then calculated based on these gain scores. Cohen (1988) identified 
the statistical levels of effect sizes as small (r < .100), medium (TOO < r < .243), and large (r > 
.371). ECAW schools demonstrated an effect size of 0.6532. 
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The change/growth rate in the performance categories of proficient and advanced, 
advanced, and unsatisfactory were also examined. ECAW schools showed a mean gain of 9.883 
in the proficient and advanced performance level, while Colorado’s mean gain was 6.777. The 
difference in means (3.106) was statistically significant at a p< 0.0122 level. This translates to 
an effect size of r = 0.5082. 

In the performance category of “advanced,” ECAW schools mean gain was 5.407. The 
Colorado gain rate was 0.866. The difference was statistically significant at a p< 0.0004 level. 
The obtained effect size for the advanced performance level was r = 0.7278. 

In the “unsatisfactory” performance category, ECAW schools showed a reduction rate in 
unsatisfactory students of 1.884%, compared to the Colorado reduction rate of 1.201. The 
difference (0.683) did not meet the standard of statistical significance (p < 0.3194). 

Considering all of the factors within the current data sets, these ECAW Member School 
cohort groups demonstrated significantly higher overall achievement and significantly higher 
rates of growth than the state of Colorado as a whole (as measured by the CSAP/TCAP Writing 
Assessment) for the years 2007 through 2013. These school cohort groups further demonstrated 
statistically significant gains in overall performance (mean scores), as well as in the performance 
categories of proficient and advanced and advanced. Further, these gains are representative of 
large effect sizes for treatment (r = .6532 overall, r = .5082 for proficient and advanced, and r = 


.7278 for advanced). 
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Directions for Further Research 

Colorado is in its final year of the CSAP/TCAP assessment. Beginning in school year 
2014-2015, the Partnership for Assessment of Readiness of College and Careers (PARCC) will 
become the new statewide assessment for Colorado. From that point forward, a new data set will 
begin to emerge, and the entire analysis process will begin anew. The same research questions 
and procedures guiding this analysis will need to be employed once again. 

The analysis in this report is predicated on two basic criteria that define the cohorts: 
ECAW teacher professional development and instructional materials. While these elements 
define these groups, several questions remain for future analysis. 

Question 1 . Did the manner in which the program was implemented in an individual school 
correlate, in any way, to the school’s assessment results? This study did not look at school or 
classroom implementation patterns in any way. Hord (1987) and her colleagues described this 
concept as the “innovation configuration.” In future studies, each school’s innovation 
configuration might be studied. Were patterns to emerge from this data, these could be useful for 
guiding schools in the direction of the most efficacious implementation practices. 

Question 2. Are there other patterns of student achievement (within this data) related to factors 
such as gender, poverty, ethnicity, English Language Learners, and/or handicapping condition? 
This data is not currently available in the Colorado Department of Education’s reporting system. 
The manner of data collection and reporting for the PARCC assessment system is not known. 
Question 3. Will the achievement gains of elementary schools be replicated in secondary 
schools? While ECAW is a K- 12 model, relatively few secondary schools have implemented. 
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